Recently, methods of performing object identification and the like using machine learning are being studied. As one of the machine learning, a deep learning, which uses a neural network with several hidden layers between an input layer and an output layer, shows high recognizing performance.
And, the neural network using the deep learning generally learns through backpropagation using losses.
In order to perform learning of such a deep learning network, training data in which tags are added to individual data points by labelers are needed. Preparing this training data (i.e. classifying the data correctly) can be very labor-intensive, expensive and inconvenient, especially if a large amount of training data is to be used and if the quality of the data pre-preparation is not consistently high. Conventional interactive labeling can be computationally expensive and fail to deliver good results.
Therefore, in recent years, auto-labeling which adds tags, i.e., labels, to a training image using a deep learning-based auto labeling device is performed, and inspectors inspect auto-labeled training image to correct the tags or the labels.
However, in such a conventional method, a throughput of the inspectors is low as compared with the throughput of the auto-labeling device. Therefore, it takes a long time to generate true labels for entire training images. In order to improve the overall throughput, the number of the inspectors must be increased, but this also increases the cost.
Also, it is difficult to acquire a large number of the inspectors skilled enough to handle the throughput of the auto-labeling device.